Skip to content

Phase 4: groups, encryption, and history - #191

Merged
dezren39 merged 27 commits into
mainfrom
phase-4
Oct 12, 2026
Merged

dezren39 merged 27 commits into
mainfrom
phase-4

Conversation

@dezren39

Copy link
Copy Markdown
Member

Phase 4 of the id design: the DataScheme adapter, scheme v1 with the version bound to the signed policy, recent:N and full history with lazy full history from a member, chunked welcomes, the heads rule with a cap of 32, rotation only on removal and update, PCS recovery as remove and re-add, local warnings at 96/128/1000 and on churn, the scheme none notice, and the update cost benchmark.

Record: doc/2026-10-12T14-33-36Z_phase_4_groups_encryption/

Full run on the merged tree: 1418 passed, 9 ignored, 1 failed. The failure, serve_tests::test_world_module_shared_over_iroh_and_web_presentations, also fails on the phase-3 tip, so this predates Phase 4 (#190).

Open, in issues: #186 held-message cap, #187 stalled delivery on missing pre-join controls, #188 flaky sync test under load, #189 joiner epoch gap and the heads merge above 32, #190 the serve stall.

Not run here: just crash-gate and the nix checks.

Adds src/encryption/: a Session per device and group over p2panda-encryption
(git rev 3e84cac, data_scheme on, test_utils and message_scheme off). Create,
add, remove, update, send and process. Membership is the member set the group
model holds after the event; after each membership change the crate's roster
is checked against it. Adds call Dcgka::add with the full bundle and run the
local steps themselves. Application messages whose group secret the device
does not hold are withheld before the crate sees them.

The orderer and the roster are our own, since the crate's MessageOrderer and
TestDgm are test-only.
… fix [P4.2]

Vendors p2panda-encryption at rev 3e84cac under pkgs/id/vendor, as the
p2panda-sync copy was in P3.2. The copy carries the S3 patch: the joiner
applies its own add in process_welcome, so a welcome's roster that predates
the add no longer leaves the joiner out, and EncryptionGroup::add_from_join.
The fix is ours, not upstream (owner decision). Each change is recorded in
vendor/README.md.

The fork's own test fails on the stock crate (the joiner's roster is
[0, 1, 2]) and passes with the patch. The adapter's joiner tests failed the
same way before the switch to the fork.
Session::add_from_join welcomes a joiner with no earlier secret (the fork's
EncryptionGroup::add_from_join) and follows it with an immediate update, so
the group has a secret the joiner learns. The joiner does not read a message
sent before its join: process withholds an application message whose group
secret the device does not hold, and gives the crate only the rest.

Tests: a joiner cannot read a message from before its join, nor one sent
between the add and the update, and reads one sent after the update. A
property over random joins (from_join and full add), removals, updates and
sends checks that members agree on the roster and epoch, read the newest
message, and that removed members are refused it.
A welcome carries the secrets its history mode allows: every secret for full, the newest N for recent:N. The adapter cuts the bundle before Dcgka::add, so the crate never sees a secret outside the window. Presets are 32 and 128. A world sets its mode through the policy's History::Recent.
The fork encodes the two byte fields as serde_bytes and writes the bundle
sorted by (timestamp, id). The version is the group policy's scheme field.
The session negotiates it at start and refuses a policy it does not speak,
so a stock group is refused, not migrated. Each envelope carries the version
in its header and in every direct message, and the session checks both before
the crate reads a message, so a relay cannot downgrade.
The crate fails a whole call on a group secret it does not hold, and a welcome that receives such a message fails with it. Round 2 saw that in 80 of 120 runs. The adapter now keeps every application message it cannot read in arrival order, outside the crate. A welcome or a later secret releases the messages it unlocks, and the call that does so returns them. The reproduction is a test: the crate refuses 6 of 18 welcomes, exactly where the window leaves an epoch out.
A joiner that holds messages it cannot read asks a member for the secrets they need. The member answers only if both it and the requester are in its roster. The secrets are sealed with HPKE to the requester's identity key, with the request as associated data, so an answer opens only for its request. A request with no answer ends in a timeout, not a hang.
The session takes the group's GroupView from groups::fold, and its members
are the roster the crate must hold after each local membership change. A
mismatch is a refusal, and the session is poisoned rather than kept with a
roster the fold does not give. A view whose policy names another scheme is
refused before the crate runs.
… budget [P4.4]

A named struct costs 17 bytes per direct message in CBOR (a map header and two
text keys). A pair costs 2, so the header copy (8 bytes) and one welcome copy
(2 bytes) come to 10 bytes.
A welcome above one world frame is split into chunks. Each chunk carries its position, the total, the digest of the whole and a check on its own bytes. The reassembly refuses an altered chunk as it arrives, a chunk out of sequence, a chunk of another welcome, and a whole that does not match its digest. The envelope is decoded only once the whole checks out. A chunk's payload is the frame less a 256-byte header budget (1 MiB frame: 1,048,320 bytes).
Two admins adding members from the same view, with a third member
ordering the two adds, left the joiner with an empty roster: the
orderer delivered an add that was ready before the joiner's welcome,
and the crate applied it before the joiner was welcomed.

The orderer now delivers a welcome after its own past and before any
other pending message. A welcome that is not yet ready holds delivery.

Properties (16 cases each): concurrent updates and sends, and a
concurrent add with sends. Each member hears the others' changes in
its own shuffled order. Every member must end with the same roster and
epoch set, and read every send by another member.

Open: an update made without knowing a concurrent add never reaches
the joiner, so the joiner holds one epoch fewer. The shape is pinned by
test_open_a_joiner_misses_an_update_made_without_knowing_it.

Adds Scheme::epoch_set, the IDs of every secret a device holds.
A control message now names the heads it can see: the control messages
that no delivered or published message names. It always names its
author's own latest control message, even when a later message covers
it. Application messages name the same parents.

The orderer used to name the latest control message of each author.
That list kept growing with the number of authors, and went past 32
parents in a group of 33, with no check. It also named messages that a
later head already covered.

A publish that would name more than 32 parents is refused with
SchemeError::TooManyParents. The check runs before the crate call, so
the refusal leaves the session as it was. Nothing is dropped from the
parents. The merge in the plan (P4.7, N22) is not built, so a group
with more than 32 heads cannot publish until a member sees fewer.

Tests: the head names only the heads (red before), the author's own
latest is named when a later message covers it, 32 parents are
accepted, and 33 are refused for an update and a send with the state
kept.
Record P4.7 as done (6001cfd, e1fee2c): the heads rule with a cap of 32 and refusal over it, the welcome-first delivery fix, the concurrency properties, and the open epoch gap for a concurrent update and add.
@dezren39
dezren39 requested a review from a team as a code owner October 12, 2026 16:18
@dezren39
dezren39 merged commit adc1cad into main Oct 12, 2026
3 of 4 checks passed
@dezren39
dezren39 deleted the phase-4 branch October 12, 2026 16:18
@coderabbitai

coderabbitai Bot commented Oct 12, 2026

Copy link
Copy Markdown

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0c380b64-7f2c-4994-a2b6-9e2b7bfde872

📥 Commits

Reviewing files that changed from the base of the PR and between c12448c and b831748.


⛔ Files ignored due to path filters (1)
  • pkgs/id/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (72)
  • doc/2026-10-11T11-08-42Z_design_remaining_work/plan-phases.md
  • doc/2026-10-12T14-33-36Z_phase_4_groups_encryption/2026-10-12T14-33-36Z_phase_4_groups_encryption.md
  • pkgs/id/Cargo.toml
  • pkgs/id/src/encryption/history.rs
  • pkgs/id/src/encryption/mod.rs
  • pkgs/id/src/encryption/notice.rs
  • pkgs/id/src/encryption/ordering.rs
  • pkgs/id/src/encryption/roster.rs
  • pkgs/id/src/encryption/scheme.rs
  • pkgs/id/src/encryption/tests.rs
  • pkgs/id/src/encryption/warnings.rs
  • pkgs/id/src/encryption/welcome.rs
  • pkgs/id/src/groups/policy.rs
  • pkgs/id/src/lib.rs
  • pkgs/id/vendor/README.md
  • pkgs/id/vendor/p2panda-encryption/Cargo.toml
  • pkgs/id/vendor/p2panda-encryption/LICENSES/Apache-2.0.txt
  • pkgs/id/vendor/p2panda-encryption/LICENSES/MIT.txt
  • pkgs/id/vendor/p2panda-encryption/README.md
  • pkgs/id/vendor/p2panda-encryption/src/crypto/aead.rs
  • pkgs/id/vendor/p2panda-encryption/src/crypto/hkdf.rs
  • pkgs/id/vendor/p2panda-encryption/src/crypto/hpke.rs
  • pkgs/id/vendor/p2panda-encryption/src/crypto/mod.rs
  • pkgs/id/vendor/p2panda-encryption/src/crypto/rng.rs
  • pkgs/id/vendor/p2panda-encryption/src/crypto/secret.rs
  • pkgs/id/vendor/p2panda-encryption/src/crypto/sha2.rs
  • pkgs/id/vendor/p2panda-encryption/src/crypto/x25519.rs
  • pkgs/id/vendor/p2panda-encryption/src/crypto/xchacha20.rs
  • pkgs/id/vendor/p2panda-encryption/src/crypto/xeddsa.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/data.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/dcgka.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/group.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/group_secret.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/mod.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/test_utils/dcgka.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/test_utils/dgm.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/test_utils/mod.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/test_utils/network.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/test_utils/ordering.rs
  • pkgs/id/vendor/p2panda-encryption/src/data_scheme/tests.rs
  • pkgs/id/vendor/p2panda-encryption/src/key_bundle/key_bundle.rs
  • pkgs/id/vendor/p2panda-encryption/src/key_bundle/lifetime.rs
  • pkgs/id/vendor/p2panda-encryption/src/key_bundle/mod.rs
  • pkgs/id/vendor/p2panda-encryption/src/key_bundle/prekey.rs
  • pkgs/id/vendor/p2panda-encryption/src/key_manager.rs
  • pkgs/id/vendor/p2panda-encryption/src/key_registry.rs
  • pkgs/id/vendor/p2panda-encryption/src/lib.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/dcgka.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/group.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/message.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/mod.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/ratchet.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/test_utils/dcgka.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/test_utils/dgm.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/test_utils/mod.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/test_utils/network.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/test_utils/ordering.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/tests/concurrency.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/tests/group_operations.rs
  • pkgs/id/vendor/p2panda-encryption/src/message_scheme/tests/mod.rs
  • pkgs/id/vendor/p2panda-encryption/src/ordering.rs
  • pkgs/id/vendor/p2panda-encryption/src/test_utils.rs
  • pkgs/id/vendor/p2panda-encryption/src/traits/dgm.rs
  • pkgs/id/vendor/p2panda-encryption/src/traits/key_bundle.rs
  • pkgs/id/vendor/p2panda-encryption/src/traits/key_manager.rs
  • pkgs/id/vendor/p2panda-encryption/src/traits/key_registry.rs
  • pkgs/id/vendor/p2panda-encryption/src/traits/message.rs
  • pkgs/id/vendor/p2panda-encryption/src/traits/mod.rs
  • pkgs/id/vendor/p2panda-encryption/src/traits/ordering.rs
  • pkgs/id/vendor/p2panda-encryption/src/two_party/mod.rs
  • pkgs/id/vendor/p2panda-encryption/src/two_party/two_party.rs
  • pkgs/id/vendor/p2panda-encryption/src/two_party/x3dh.rs

 ______________________________________
< Hold on, let me under-engineer this! >
 --------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR

🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant